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Building  the  Analytic  Foundation  for  DIS  TOOL: 

A  Planning  Tool  for  Users  of  Distributed  Interactive  Simulation  (DIS) 


I.  Introduction 

The  press  of  technology  and  a  reduction  in  resources  has  led  the  U.S.  Army  to  develop 
a  technology-based  training  strategy.  The  premier  example  of  this  strategy  is  the  distributed 
interactive  simulation  (DIS)  environment.  The  DIS  environment,  an  outgrowth  of  the  DARPA 
sponsored  Simulation  Networking  (SIMNET)  technology  development  effort,  is  currently  being 
used  to  support  combined  arms  training  as  well  as  combat  development  efforts.  DIS  provides 
an  effective  and  cost  efficient  means  for  supporting  both  training  and  combat  developments  in 
part  through  the  use  of  computer  generated  forces  (CGF)  to  serve  as  either  opposition  forces 
(OPFOR)  or  adjacent  or  supporting  elements  of  friendly  forces. 

The  Army  Science  Board  (ASB)  Summer  Study  on  Army  Sunulation  Strategy  (1991) 
acknowledged  the  power  of  the  technology  push  on  training  ^sterns  by  arguing  that  the  Army 
aggressively  adopt  the  electronic  battlefield  for  traimng,  with  emphasis  on  achieving  a  combined 
arms  capability.  The  ASB  concluded  that  the  use  of  simulation  is  on  the  increase  and  serves  a 
vital  role  in  preparing  the  Army  for  the  21st  century.  The  increased  use  and  importance  of 
simulation  was  attributed  to  constrained  funding,  test  range  and  training  space  limitations,  the 
high  cost  of  hardware  demonstrations  of  new  systems,  and  new  simulation  capabilities.  DIS 
capabilities  afford  a  wide-range  of  opportunities  to  address  the  needs  of  various  Army 
communities  interested  in  combat  development,  system  acquisition,  test  and  evaluation,  and 
training. 

Those  using  simulation  capabilities  such  as  DIS  are  doing  so  in  an  attempt  to  keep  pace 
with  a  rapid  evolution  of  electronic  technologies.  These  technologies  are  being  configured  into 
devices  and  weapon  systems  intended  to  enhance  combat  performance;  however,  they  introduce 
new  system  development,  training,  and  test  and  evaluation  requirements  which  must  be  addressed. 
Evidence  of  the  trend  towards  using  DIS  to  meet  the  Army's  changing  requirements  is  provided 
by  the  Army’s  Modernization  Plan  for  Training  (Department  of  the  Army,  1993b)  which  calls  for 
a  move  toward  greater  use  of  training  aids,  devices,  simulators  and  simulations  (TADSS).  Key 
aspects  of  the  modernization  plan  call  for  technology-based  elements,  including  a  ^stem  which 
provides  a  common,  consistent  and  accredited  simulated  representation  of  the  combined  force 
battlefield  (DIS).  According  to  the  modernization  plan,  "DIS  enhances  warfighting  capabilities; 
the  fielding  of  superior  weapons  systems;  and,  early  development  of  doctrine  and  tactics,  force 
development,  rapid  prototyping,  and  the  evaluation  of  tech  base  concepts..." 

A  requirement  generated  by  the  use  of  DIS  is  the  specification  of  performance  variables 
which  address  not  only  the  outcomes  of  performance,  the  courses  of  action  taken,  and  the 
conditions  under  which  the  mission  was  performed  but  also  the  cognitive  and  perceptual  variables 
of  the  participants  that  influence  performance.  Currently,  there  is  little  understanding  of  the 
relationship  between  human  cognitive  and  perceptual  variables  and  combat  performance.  To  date, 
human  perceptual  and  cognitive  domains  have  not  been  programmed  into  DIS  for  CGF  modeling, 
nor  classified  into  a  taxonomy  of  human  performance  variables  relevant  to  combat.  Generally 
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speaking,  most  attention  has  been  focused  on  the  modeling  of  physical  attributes  and 
characteristics  in  the  creation  of  the  DIS  environments.  Thus,  this  paper  addresses  the  need  for 
a  DIS  TOOL  to  facilitate  user  access  to  a  taxonomy  of  human  performance  for  perceptual, 
cognitive,  and  motor  processes  which  occur  under  combat  situations.  DIS  TOOL  will  support 
the  identification  of  measures  of  performance  (MOPs)  and  measures  of  effectiveness  (MOEs)  that 
win  satisfy  the  anticipated  needs  of  a  wide  variety  of  Army  communities;  especiaUy  those 
interested  in  training  development,  combat  development,  test  and  evaluation,  and  CGF 
development. 

Performance  measures  are  at  the  heart  of  empiricaUy  deriving  the  value  and  benefit  of  new 
systems  and  training  development  efforts.  Therefore,  performance  assessment  with  reUable  and 
vaHd  measures  should  be  a  chief  concern  of  any  DIS  user.  Assessments  and  their  measures 
provide  the  basis  for  studies  of  effectiveness  and  efficiency  that  allow  for  informed  trade-off 
decisions.  The  derivation  of  the  performance  measures  in  a  simulation  environment  is  a  complex 
undertaking  and  must  account  for  multidimensional  performance  parameters.  The  development 
of  DIS  TOOL  is  critical  to  users  of  DIS  technology  who  recognize  the  need  to  go  beyond  the 
domain  of  physical  variables  (e.g.,  distance  travelled)  to  study  how  behaviors  that  are  primarily 
perceptual  or  cogmtive  in  nature  relate  to  combat  performance. 


II.  Backaround 


Faced  with  the  changing  battlefield  dynamics  of  the  next  century  and  the  advent  of  the 
information  age  of  technology,  soldiers  wiU  be  waging  an  information  war.  DIS  will  play  a 
pivotal  role  in  helping  the  Army  speed  the  acquisition  cycle  of  emerging  warfighting  technologies 
by  supporting  their  design,  development,  testing,  and  personnel  training  through  rapid 
prototyping.  To  ensure  that  the  Army  is  prepared  to  win  the  information  war.  Army  communities 
must  account  for  the  relationship  of  perceptual  and  cogmtive  processes  to  battlefield  behavior  by 
exploiting  ciurent  DIS  features  and  developing  important  enhancements  such  as  DIS  TOOL. 

Cmrent  DIS  features  include: 

•  Vertical  and  horizontal  network  distribution  for  collective  training  and  long-haul 
networking. 

•  Selective  functional  fidelity  which  permits  the  prioritization  of  functional 
requirements 

•  Virtual  prototyping  which  supports  the  use  of  modeling  and  simulation  to  assist 
in  the  system  acquisition  process. 

•  Real-time,  man-in-the-loop  simulation. 
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•  Seamless  simulation  which  permits  networked  interaction  between  synthetic  and 
real-world  environments. 

•  Varying  threat  environments  supported  by  semi-automated  forces. 

•  Variety  of  terrain  data  bases  to  support  contingency  missions. 

These  DIS  features  can  be  used  to  develop  new  assessment  strategies  and  innovative 
efforts  directed  at  planned  activities  such  as  constructive  simulation  exercises;  doctrine  and  tactics 
development;  command  and  unit  training  (including  combined-arms  training);  and  operational 
plaiming  and  rehearsal.  However,  DIS  users  need  an  instrument  which  will  guide  their  planning 
efforts  and  ensure  that  the  underlying  percepmal  and  cognitive  processes  that  drive  battlefield 
performance  will  be  adequately  assessed.  To  this  end,  DIS  TOOL  will  play  an  important  role 
in  aiding  combat  developers,  training  developers,  CGF  designers  and  test  and  evaluation 
communities.  DIS  TOOL'S  primary  contribution  will  be  the  identification  of  behavioral 
components  of  performance.  DIS  TOOL  will  support  accurate  unit  performance  assessment  by 
providing  a  taxonomy  of  human  performance  variables  that  encompasses  perceptual  and  cognitive 
processes.  Thus,  DIS  users  who  have  an  interest  in  mission  performance  which  extends  beyond 
measurement  approaches  aimed  only  at  physical  behaviors  stand  to  benefit  substantially  from  the 
development  of  DIS  TOOL. 

Simulation  offers  a  cost-effective  strategy  for  providing  Army  communities  a  variety  of 
conditions  under  which  to  conduct  research  and/or  training.  However,  as  digitization  of  the 
battlefield  evolves,  followed  by  the  information  war,  there  is  a  growing  need  for  a  greater  range 
of  performance  measures  suitable  for  a  variety  of  purposes  related  to  developing  and  assessing 
training,  equipment,  or  CGFs  that  will  adequately  support  new  battlefield  warfighting  strategies. 
A  logical  extension  of  pioneering  work  done  by  the  Army  Research  Institute  (ARI),  DIS  TOOL 
will  help  fill  the  current  performance  measurement  void. 

In  a  number  of  ways,  ARI  has  laid  the  foundation  for  measuring  unit  mission  performance 
and  for  recognizing  the  need  to  extend  traditional  measurement  approaches  to  accommodate 
anticipated  changes  in  training  requirements  and  technologies.  First,  ARI  conducted  research 
focused  on  unit  performance  at  the  National  Training  Center  (NTC)  which  resulted  in  a  systems 
model  specifying  both  outcome  and  process  measurement  systems  (Lewman,  et  al.,  1988).  Next, 
this  work  was  built  upon  to  create  the  Unit  Performance  Assessment  System  (UPAS)  which 
supports  unit  performance  measurement  and  training  feedback  in  DIS  environments  by  allowing 
for  the  measurement  of  physical  system  variables,  such  as  position  of  systems,  firing  activities, 
casualties  suffered,  and  ammunition  expenditures  (Meliza,  Bessemer,  Burnside,  &  Schlechter 
(1992).  Additional  advances  in  performance  measurement  have  been  made  by  the  ARI  Fort  Knox 
Field  Unit  as  part  of  the  Combat  Vehicle  Command  and  Control  (CVCC)  program.  This  effort 
included  both  automated  data  capture  similar  to  UPAS,  as  well  as  observational  data  focused  on 
command,  control,  and  communication  (C).  The  CVCC  research  (e.g.,  Leibrecht,  et  al.,  1993) 
utilized  innovative  approaches  to  performance  measurement  by  addressing  how  the  situational 
awareness  of  the  participants  was  impacted  by  automated  C  devices.  Thus,  the  development  of 
performance  measurement  capabilities  such  as  UPAS  and  the  research  associated  with  the  CVCC 
program  have  laid  the  foundation  for  the  development  of  DIS  TOOL  which  will  address  the  full 
complement  of  cognitive  and  perceptual  variables  that  play  a  role  in  combat  performance. 
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Qearly,  simulation-based  training  and  assessment  methods  require  the  appropriate 
measurement  and  planning  tools  to  support  effective  use  of  DIS  environments.  A  major 
proponent  of  DIS  technology  that  will  benefit  substantially  from  the  development  of  DIS  TOOL 
is  the  Army  Battle  Labs.  Proposed  by  Gen.  Franks,  battle  labs  will  focus  on  emerging 
technologies  and  changes  in  tactics  and  doctrine.  The  six  battle  labs  will  conduct 
experimentation  using  DIS  capabilities  to  support  the  determination  of  requirements  driven  by 
the  future  battlefield  (Department  of  the  Army,  1993a).  These  battle  labs  will  provide  different 
Army  communities  the  opportunity  to  integrate  their  various  knowledge  areas  with  industry, 
resulting  in  a  powerful  research  and  development  network.  The  research  areas  of  each  of  the  six 
battle  labs  are  listed  in  Table  1  and  reflect  the  Army's  clear  need  to  reach  beyond  traditional 
measurement  strategies  that  primarily  focus  on  the  physical  dimension  of  performance. 


Table  1 

Battle  Lab  Research  Areas 


BATTLE  COMMAND  (FoRT  LEAVENWORTH, KS) 

C2  movement  ability 

•  Combat  Service  Support  battlefield  automation 

Horizontal  integration  of  digitized  information 

EARLY  ENTRY,  LETHALITY,  AND  SURVIVABILITY  (FORT  MONROE,VA) 

Lethality,  survivability,  mobility,and  sustainability  of  Early  Entry  Force 
Intelligence  Preparation  of  the  Battlefield 

Armored  force  deployability 

DEPTH  AND  SIMULTANEOUS  ATTACK  (Fort  Sill,  OK) 

Accuracy  of  attack  systems 

Enemy  force  at  maximum  depth 

Near-real-time  intelligence  and  targeting  information 

Intelligence  and  electronic  warfare  with  attack  systems  in  near-real-time 
to  optimize  targeting 

DISMOUNTED  BATTLE  SPACE  (FORT  Benning,  GA) 

Night  fighting  for  combined  arms,  combat  support,  and  combat  services 
support 

Target  acquisition  across  combined  arms  forces 

Soldier  survivability  through  signature  reduction,  individual  sensors,  and 
lightening  the  ioad 

Lethality  of  weapons  for  direct  and  indirect  fire 

MOUNTED  BATTLE  SPACE  (Fort  Knox,  KY) 

Situational  awareness  and  target  hand-off 

Survivability  of  the  mounted  force 

Armored  threats  in  all  conditions  with  increased  probability  of  success 

COMBAT  SERVICE  SUPPORT  (Fort  Lee,  VA) 

Soidier  and  system  sustainment 

Asset  visibility  for  all  classes  of  supply 

Logistics  communication  and  automation 
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DIS  TOOL  will  be  a  performance  measurement  asset  to  Army  communities  such  as  the 
battle  labs  by  assisting  in  the  following  activities: 

•  Identifying  and  characterizing  tasks  and  behaviors  that  can  be  performed  or 
simulated  based  on  Critical  Combat  Functions  (CCFs),  echelon,  and  METT-T 
conditions; 

•  Guiding  users  in  specifying  performance  requirements  for  specific  applications; 

•  Identifying  appropriate  variables  to  measure  performance  of  relevant  tasks  and 
behaviors; 

•  Allowing  users  to  select  appropriate  variables  and  retrieve  previously  used 
measures  and  define  new  measures; 

•  Allowing  users  to  identify  gaps  in  behaviors  that  can  be  simulated  or  measured  in 
existing  systems. 

In  sum,  DIS  TOOL  will  ^pport  the  identification  of  TADSS-based  measures  which  have 
a  high  utility  for  various  Army  communities.  DIS  TOOL  will  save  users  from  identifying  and 
cataloguing  aU  potential  measures  which  would  be  an  extremely  large  and  resource  intensive  task. 
By  providing  an  operational  firamework  of  performance  measurement  which  covers  a 
comprehensive  domain  of  multidimensional  variables,  DIS  TOOL  will  allow  users  to  plan  DIS 
exercises  using  select  measures  which  have  doctrinal  soundness  and  compatibility  with 
operational/field  measures. 


III.  DIS  TOOL  Overview 


A.  Vision  of  DIS  TOOL.  Army  development  communities  must  now  do  more  (use 
DIS)  with  less  (shrinking  budgets,  persoimel  downsizing)  and  achieve  higher  standards  (satisfy 
requirements  for  emerging  doctrine,  new  threats,  and  complex  equipment).  The  development  of 
a  taxonomy  of  human  performance  variables  spanning  cognitive  and  perceptual  dimensions  is  an 
important  first  step  towards  the  development  of  tools  that  can  assist  different  Army  communities 
in  the  optimal  utilization  of  ^nthetic  environments  like  DIS.  A  major  obstacle  that  will  be 
overcome  by  DIS  TOOL  relates  to  the  fact  that  most  DIS  research  has  focused  on  individual 
performance  and  individual  attributes.  The  DIS  TOOL  taxonomy  focuses  primarily  on  collective 
training  and  thus  is  concerned  with  unit  as  well  as  individual  performance.  At  present,  definitive 
research  has  not  been  performed  that  maps  individual  characteristics  to  groups  or  vise  versa.  DIS 
TOOL  will  make  a  significant  contribution  to  imderstanding  this  issue  in  that  it  will  provide  a 
taxonomy  which  spans  individual  soldier  through  crew  to  battalion  performance. 

DIS  TOOL  win  provide  trainers,  combat  developers,  test  and  evaluation  personnel, 
equipment  designers,  software  engineers,  and  modelers  with  a  powerful  tool  to  use  in  planning 
doctrinally  sound  electronic  exercises.  The  completed  taxonomy  will  demonstrate  a  direct  linkage 
to  the  doctrinal  sources  used  by  soldiers  as  they  train.  To  accomplish  this,  a  key  challenge  was 


5 


locating  a  doctrinally  sound  analytic  struchire  which  provided  the  appropriate  level  of  detail. 
That  is,  the  analytic  structure  had  to  be  sufficiently  detailed  to  support  performance  measurement 
while  also  providing  a  high  organizational  structure  to  sustain  focus  and  be  clear  about  inter¬ 
relations  between  tasks  such  as  synchronization  and  coordination.  The  CCFs  will  provide  a 
useful  framework  in  this  respect.  The  CCFs  are  at  a  level  of  aggregation  between  tasks  and  the 
Battlefield  Operating  Systems  (BOS)  contained  within  the  Blueprint  of  the  Battlefield 
(Department  of  the  Army,  1991).  Further,  they  provide  direct  linkages  across  echelons  in  terms 
of  task  dependencies.  Finally,  the  CCFs  specify  key  players,  procedures,  and  information  flows. 
As  such,  they  form  the  most  complete  specification  of  battlefield  functionality  at  battalion  and 
below  developed  to  date.  The  CCFs  are  described  in  detail  in  Mullen  (in  preparation). 

B.  DIS  TOOL’S  linkage  to  CCFs.  DIS-TOOL  is  intended  to  provide  the  user  with 
the  information  needed  to  construct  exercises  and  measxirement  approaches  that  address  the 
concerns  of  the  user.  The  structure  of  DIS-TOOL  is  plaimed  to  be  highly  flexible  and  relates  to 
a  number  of  kinds  of  higher  organization  used  in  the  military  setting  for  tasks  and  activities. 
This  type  of  ^stem  is  needed  to  address  the  concerns  of  the  various  Army  communities.  To 
incorporate  the  needs  of  these  communities,  a  general  organizational  framework  for  military 
combat  activities  must  be  used.  As  a  trainer's  tool,  DIS-TOOL  should  relate  to  these  or  other 
types  of  organization  used  in  the  completion  of  military  combat  activities.  For  example,  past  DIS 
exercises  have  used  BOSs  as  an  organizational  structure  for  addressing  key  research  issues. 
While  BOSs  have  been  useful  they  are  also  at  an  extremely  broad  level,  which  makes  addressing 
specific  research  questions  difficult.  The  complexity  of  the  interrelationships  that  occur  in 
combat  situations  has  lead  to  the  development  of  CCFs.  The  CCFs  are  compatible  with  the 
BOSs  but  break  down  the  combat  operations  process  into  smaller  units.  This  allows  for  the 
examination  of  more  specific  relationships  and  the  ^nchronization  of  different  functions.  Thus, 
the  CCFs  contribute  a  rich  structure  from  which  to  develop  and  address  research  questions  which 
may  be  aimed  at  a  variety  of  issues  relevant  to  doctrine  and  tactics. 

CCFs  provide  a  suitable  framework  for  DIS  TOOL  by  examining  the  complex  interlocking 
activities  that  occur  during  a  combat  situation  and  providing  a  level  of  description  that  is 
intermediate  in  focus.  These  features  allow  the  CCFs  to  play  a  critical  role  in  building  a  bridge 
between  BOSs  and  Army  Mission  Training  Plans  (AMTPs).  In  addition,  CCFs  provide  a  way 
to  incorporate  the  doctrinal  sources  such  as  Army  Training  and  Evaluation  Programs  (ARTEP) 
as  well  as  AMTPs  with  the  functional  requirements  of  combat  power.  Using  CCFs  as  the 
organizational  framework  for  DIS  TOOL  has  a  number  of  practical  applications.  First,  training 
that  is  based  on  functional  requirements  will  be  highly  useful  because  it  will  tend  to  focus  on  a 
broader  range  of  requirements  necessary  for  successful  task  completion.  Second,  training  that 
can  be  linked  to  outcome  measures  as  well  as  lessons  learned  demonstrates  the  utility  of  the 
training  involved. 

C.  CCF  structure.  As  previously  mentioned,  CCFs  can  be  interrelated.  For 
instance,  CCF  15  (coordinate,  synchronize,  and  integrate  fire  support),  is  related  to  CCFs  6  (fire 
support  and  maneuver),  16/17  (take  active/passive  air  defense  measures)  and  25  (provide 
operations  security).  These  interrelationships  are  important  to  consider  in  that  they  point  the 
direction  toward  desirable  outcomes  that  affect  the  battle.  As  an  example,  a  training  developer 
can  refer  to  the  task  analysis  for  CCFs  16/17  (Jarrett,  in  progress)  to  find  that  CCFs  16/17  link 
to  CCF  15  in  terms  of  the  tasks  associated  with  the  integration  of  fires  needed  to  attack  enemy 
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helicopter  positions.  In  addition  to  specifying  interrelationships  between  CCFs,  the  task  analysis 
will  also  provide  DIS  users  with  a  description  of  how  each  CCF  relates  to  a  particular  BOS; 
specify  the  outcomes  and  purposes  of  the  CCF;  provide  a  task  flow  by  task  force  battle  phase 
plan,  prepare  and  execute  diagrams;  list  key  participants  by  task;  list  key  inputs;  and  specify  a 
task  list  summary  and  full  task  lists  for  the  planning,  preparing,  and  execution  of  missions.  This 
information  is  structured  in  such  a  way  that  specific  components  (e.g.,  interrelationships  between 
CCFs)  may  be  considered  in  isolation.  Thus,  the  CCF  structure  enables  an  individual  to  consider 
a  number  of  different  information  categories  without  becoming  mired  in  the  detailed  tasks  of  any 
particular  CCF. 

DIS  TOOL,  in  conjunction  with  the  CCFs,  will  provide  a  means  to  organize  the  wide 
range  of  information  that  is  available  to  the  user.  By  breaking  down  the  battlefield  into 
functional  units,  tasks  associated  with  those  functional  units  will  become  clear.  Table  2  shows 
how  each  of  the  39  CCFs  are  embedded  within  a  particular  BOS  and  illustrates  the 
complementary  nature  of  the  Blueprint  of  the  Battlefield  (BOB)  to  the  CCFs.  CCF  descriptions 
presented  below  are  taken  from  Mullen  and  Kastanek  (in  progress). 

Intglliggnge  BOS 

These  CCFs  include  aU  functions  related  to  intelligence  planning,  collection,  processing, 
and  dissemination.  Conduct  Intelligence  Planning  (CCF  1)  primarily  focuses  on  the  Intelligence 
Preparation  of  the  Battlefield  (IPB)  and  includes  reconnaissance  and  surveUlance  plan,  threat 
templates,  and  terrain  and  weather  analyses.  Collect  Information  (CCF  2)  focuses  on  gathering 
information  from  various  sources  and  also  includes  managing  and  processing  these  activities. 
Process  (CCF  3)  and  dissemination  (CCF  4)  of  intelligence  involve  the  conversion  of  intelligence 
into  a  usable  format  and  the  transmission  of  this  information  to  appropriate  members  of  the 
combined  arms  team.  Together,  these  CCFs  cover  aU  activities  related  to  intelligence  tasks. 
These  CCFs  provide  the  activities  relevant  to  acquiring,  analyzing,  and  using  knowledge  of  the 
enemy,  weather,  and  terrain  to  give  the  commander  with  the  necessary  inputs  in  planning, 
preparing,  and  executing  combat  operations.  An  important  aspect  of  these  CCFs  is  that  they  are 
continuous  throughout  all  phases  of  the  battle. 

Maneuver  BOS 

The  CCFs  in  this  group  involve  the  use  of  direct  fire  weapons,  platforms,  and  systems 
through  movement,  fire,  and  maneuver  to  achieve  a  position  of  advantage  in  relation  to  enemy 
ground  forces.  Two  CCFs  are  grouped  here:  conduct  tactical  movement  (CCF  5)  and  engage 
enemy  with  direct  fire  and  maneuver  (CCF  6).  These  CCFs  are  designed  to  cover  the  positioning 
of  direct  fire  weapons  systems  as  well  as  using  direct  fire  and/or  close  combat  to  destroy  or  make 
the  enemy  withdraw.  These  CCFs  include  tasks  involved  in  the  dissemination  of  OPORDs, 
preparation  and  rehearsal  of  battle  plans  and  maneuver,  protection  and  control  of  terrain,  and 
consolidation  and  reorganization  tasks. 
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Table  2 

Index  of  Critical  Combat  Functions  by  BOS 


INTELLIGENCE 

MANEUVER 

AIR  DEFENSE 

FIRE  SUPPORT 

MOBILITY  and 
SURVIVABILITY 


COMMAND  and 
CONTROL 


COMBAT  SERVICE 
SUPPORT 


(1)  Conduct  Intelligence  Planning 

(2)  Collect  Infonnation 

(3)  Process  Infonnation 

(4)  Disseminate  Intelligence 

(5)  Tactical  Movement 

(6)  Engage  Enemy  with  Direct  Fire  and  Maneuver 

(16)  Active  Air  Defense  Measures 

(17)  Passive  Air  Defense  Measures 

(7)  Mortars 

(8)  Artillery 

(9)  Close  Air  Support 

(10)  Conduct  Electronic  Collection  and  Jamming 

(11)  Conduct  Battlefield  PsyOps 

(12)  Employ  Chemical  Weapons 

(13)  Counter  Target  Acquisition  Systems 

(14)  Naval  Gunfire  (Deleted) 

(15)  Fire  Support  Integration 

(21)  Overcome  Obstacles 

(22)  Enhance  Movement 

(23)  Provide  Countermobility 

(24)  Enhance  Physical  Protection 

(25)  Operations  Security 

(26)  Deception 

(27)  Decontamination 

(18)  Plan  for  Combat  Operations 

(19)  Direct  and  Lead  Unit  During  Preparation  Phase 
of  Battle 

(20)  Direct  and  Lead  Unit  in  Execution  of  Battle 

(28)  Provide  Transport  Services 

(29)  Supply 

(30)  Persotmel 

(31)  Maintenance 

(32)  Health  Services 

(33)  Battlefield  Casualty  and  Excavation 

(34)  Enemy  POW  Operations 

(35)  Law  and  Order  Operations 

(36)  Civil  Affairs  Operations 

(37)  Provide  Sustainment  Engineering 

(38)  Evacuate  Noncombatants 

(39)  Provide  Field  Services 
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Fire  Support  BOS 

The  CCFs  in  this  group  involve  the  collective,  coordinated,  and  synchronized  use  of  target 
acquisition  data,  indirect  fire  weapons,  armed  aircraft,  and  other  means  against  ground  targets  in 
support  of  maneuver  force  operations  and  to  achieve  commander’s  intent  and  scheme  of 
maneuver.  These  CCFs  include  the  use  of  mortars,  field  artillery,  close  air  support,  chemical 
weapons,  and  naval  gunfire.  A  number  of  these  CCFs  involve  actions  taken  to  disrupt  enemy 
command,  control,  and  communications  or  combat  operations  such  as  electronic  collection  and 
jamming  and  the  conduct  of  battlefield  p^ops.  Most  involve  the  employment  of  different  types 
of  weapons  systems  and  the  coordination  of  those  weapons  systems. 

Air  Defense  BOS 

The  CCFs  related  to  this  BOS  involve  taking  active  and  passive  air  defense  measures. 
These  actions  are  designed  to  stop  or  reduce  the  effectiveness  of  attack  by  hostile  forces.  Active 
ak  defense  measures  include  the  use  of  air  defense  artillery,  maneuver  unit  weapon  systems  and 
the  coordination  of  these  ^sterns.  Passive  air  defense  measures  include  dispersion,  cover  and 
concealment,  and  deception. 


Command  and  Control  BOS 

These  CCFs  involve  how  the  maneuver  commander  exercises  authority  and  direction  over 
organic  and  assigned  combat  power  in  the  accomplishment  of  the  mission.  They  frequently 
involve  the  direction  and  leading  in  a  battle.  The  focus  is  on  the  planning  of  combat  operations 
and  providing  direction  and  leadership  in  preparation  and  execution  for  the  battle. 

MpMjly.^d-,Stiryiyai?iiity  PQ.S 

These  CCFs  involve  how  a  force  engages  in  maneuver,  counter  maneuver,  and  deception 
measures,  and  enhances  protection  and  security.  A  variety  of  tactics  may  be  employed  here 
designed  to  accomplish  different  goals  in  overcoming  obstacles  or  enhancing  movement.  These 
may  involve  overcoming  obstacles,  facilitating  movement  for  other  units,  conducting  deception 
operations,  or  enhancing  physical  protection  and  security. 

Combat  Service  Support  BOS 

The  majority  of  CCFs  fall  into  this  category  and  entail  the  variety  of  operations  that  are 
necessary  to  sustain  sources  and  include  logistics,  personnel  services,  and  health  services.  These 
are  activities  necessary  to  maintain  the  operations  of  the  force.  These  may  be  logistical  in  nature 
such  as  providing  transport  services  or  supply  operations,  and  maintaining  weapons  systems  and 
equipment.  Personnel  services  may  include  financial  services,  legal,  and  public  affairs.  Health 
services  include  treating  and  evacuating  casualties. 

CCFs  are  useful  because  they  are  divided  into  small  units  so  that  the  interrelationships 
of  different  functions  may  be  examined.  Therefore,  it  is  logical  to  think  that  certain  groups  of 
CCFs  will  be  critical  during  different  parts  of  the  battle  and  that  users  of  the  DIS  system  wiU 
want  to  examine  the  interrelationships  of  several  CCFs  in  constructing  an  exercise.  Thus, 
coordination  of  direct  fire  may  be  a  particularly  important  issue  and  may  be  measured  in  terms 
of  fi'atricide  incidents,  or  the  dimensions  of  timeliness  and  accuracy. 
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Key  Objectives  and  Technical  Approach 

The  development  of  DIS  TOOL  is  focused  on  two  primary  technical  objectives  as 
identified  below: 

1.  To  design  and  execute  procedures  for  developing  a  taxonomy  of  human  performance 
variables,  including  perceptual  and  cognitive  variables,  that  may  be  used  to  identify 
significant  dimensions  that  need  to  be  measured  to  enable  accurate  unit  performance 
assessment  in  the  DIS  environment.  The  taxonomy  will  be  designed  for  use  as  a  planning 
tool  that  would  assist  combat  developers,  training  developers,  CGF  designers,  and  support 
test  and  evaluation  in  the  DIS  environment. 

2.  To  demonstrate  the  utility  of  the  product  in  controlling  CGF  behavior  and  analyzing  unit 
performance.  This  objective  requires  showing  how  the  contents  of  the  taxonomy  can  be 
used  to  identify  refinements  and  enhance  the  value  of  training  feedback  and  testing  within 
the  DIS  environment. 

The  final  DIS  TOOL  will  meld  the  product  of  these  two  objectives,  resulting  in  a 
comprehensive  planning  tool  for  a  wide  variety  of  DIS  users.  The  initial  development  effort 
described  here  began  with  an  independent  but  parallel  and  related  approach  to  meeting  the  two 
objectives,  with  the  intention  of  building  a  comprehensive  taxonomic  framework  that  would 
satisfy  both  objectives.  Figure  1  illustrates  the  relationship  between  the  development  efforts 
which  to  date  have  been  primarily  analytic  in  focus.  The  current  effort  has  laid  the  ground  work 
for  satisfying  Objective  1  and  is  described  in  this  report  while  the  approach  to  satisfying 
Objective  2  may  be  found  in  Weaver  and  Mullen  (in  preparation).  The  planned  approach  utilized 
for  the  accomplishment  of  Objective  1  was  evolutionary  in  nature.  This  approach  allows  for 
continuous  refinement  and  input  so  that  the  resulting  product  incorporates  as  much  innovation 
and  creative  thinking  as  possible  given  the  time  and  resource  constraints.  While  the  DIS  TOOL 
taxonomy  requires  further  development,  testing,  modification,  and  alliance  with  Objective  2,  a 
strong  foundation  for  supporting  its  continued  development  has  been  built.  The  remainder  of  this 
section  will  present  an  overview  of 
the  technical  approach  used  to 
achieve  Objective  1  while  Chapter 
4  will  further  describe  the  tasks 
remaining  to  be  accomplished  for 
DIS  TOOL  in  order  to  complete 
this  effort  and  the  final  taxonomy. 


The  technical  approach  to 
taxonomy  development  was  based 
on  two  principal  tenets.  The  first 
is  that,  if  the  DIS  TOOL  is  to 
serve  multiple  users,  then  each 
user  group  must  be  represented 
throughout  the  development  Figure  1  Relationship  Between  DIS  TOOL  development 

process.  Thus,  an  advocate  for  efforts. 


INPUT 


PROCESS 


OUTPUT 


HUMAN 


CGF 

DESIGNER  ■ 
TAXONOMY 
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each  community  was  appointed  to  represent  the  views  of  that  community.  This  is  illustrated  in 
Figure  2,  where  five  principal  communities  have  been  represented.  In  the  performance  of  the 
advocacy  role,  each  individual  was  to  act  as  a  filter  viewing  the  tasks  and  elements  of  the 
taxonomy  from  the  perspective  of  the  purposes  appropriate  to  that  community.  The  second  tenet 
is  structural  in  nature  and  suggests  that  the  human  performance  characteristics  and  the  influences 
of  the  environmental/  contextual  variables  are  focused  on  subtasks  (products/outcomes), 
processes,  and  actions  embedded  in  the  higher  order  structure.  The  technical  approach  was  also 
shaped  by  specific  milestones  involved  in  the  initial  development  and  completion  of  the 
taxonomy.  These  milestones  are  highlighted  in  Table  3  and  described  in  BDM  Federal  (1993). 
The  current  status  of  major  milestones  is  reviewed  in  Chapter  4. 


Figure  2.  DIS  TOOL  User  Communities. 
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Table  3 

DIS  TOOL  Milestones 


PREPARE  COMPREHENSIV^ERFORMANCE  PLAN 

Conduct  Kickoff  Meeting 

Prepare  Draft  Piafi  V  ^  ^ 

Incorporate  Feedback 

DEVELOP  A  TAXdNOMY  OF  GOGNITIVE  AND  PERCEPTUAL  VARIABLES 
FOR  THE  DIS  ENVIRONMENT 

Specify  Boundary  Conditions 

Specify  Taxonomy 

Specify  Variable  Definitions 

•  ■■  ■ 

Plari  for  Taxonomy  Completion 

DEVELOP  DIS  TOOL  AND  ITS  APPLICATION  TO  CGF  AND  UNIT 
PERFORMANCE  ASSESSMENT 


Develop  Software  representation  of  the  taxonomy 
Demonstrate  exterrial  validity  of  the  taxonomy 


Taxonomy  Description 

Any  taxonomic  effort  must  consider  several  important  factors  in  its  development.  This 
is  particularly  true  for  a  system  that  is  designed  to  be  a  multipurpose  tool  in  planning  DIS 
exercises. 

The  first  consideration  is  the  purpose  that  the  taxonomy  must  serve.  The  taxonomy's 
purpose  must  be  clearly  and  explicitly  defined.  Second,  a  taxonomy  should  use  the  appropriate 
subject  matter  and  organizational  structure  for  the  subject  matter.  This  type  of  information  may 
consist  of  observed  behaviors,  hypothesized  intervening  processes,  required  abilities,  or  measured 
responses.  Third,  a  method  for  classification  must  be  developed  which  consists  of  rules  which 
determine  class  inclusion  or  exclusion.  Each  of  these  considerations  should  be  taken  into  account 
when  constructing  and  evaluating  a  taxonomy.  To  this  end,  the  taxonomy  proposed  here  is 
designed  to  deal  with  the  processes  as  well  as  overt  measurements.  This  taxonomy  is  seen  as 
related  to  other  elements  such  as  performance  dimensions,  categories  of  measures,  indicators,  and 
utility/value.  This  provides  the  opportunity  for  the  user  to  "enter"  the  taxonomy  firom  a  number 
of  different  perspectives,  based  upon  the  questions  being  asked. 

The  purpose  of  the  DIS  TOOL  taxonomy  is  to  allow  an  individual,  regardless  of  the  Army 
community  to  which  they  belong,  to  make  use  of  available  information  in  such  a  way  that  will 
optimize  the  planning  of  DIS  exercises  for  a  variety  of  users.  This  suggests  that  the  taxonomy 
must  be  flexible  in  nature  so  that  individuals  with  varying  needs  will  be  able  to  easily  use  it.  In 
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other  words,  people  will  want  to  examine  problems  from  a  number  of  different  perspectives  and 
the  taxonomic  structure  should  take  this  into  account.  For  example,  a  trainer  may  want  to 
examine  the  processes  (e.g.  perceptual,  cognitive,  or  motor)  relevant  to  effective  command  and 
control  performance.  On  the  other  hand,  a  combat  developer  may  be  interested  in  the  accuracy 
of  a  new  weapon  system  as  modeled  in  the  DIS  environment.  Thus,  the  training  developer  would 
most  likely  enter  the  taxonomy  at  a  different  location  than  the  combat  developer. 

Figure  3  presents  the  general  higher  order  structure  of  the  taxonomy.  The  highest  level 
is  mission,  followed  by  Global  METT-T,  CCF,  phase,  and  task.  Echelon  is  not  separately 
represented  as  it  is  included  in  the  CCF  structure.  These  five  levels  are  consistent  with  doctrine 
and  allow  for  the  use  of  decompositions  similar  to  those  accomplished  during  the  study  of  the 
CCFs.  Below  the  task  level,  the  intervention  of  human  characteristics  and  the  factors  of  METT-T 
are  reflected.  For  example  the  cues  that  elicit  specific  behaviors  come  from  the  local  METT-T. 
The  introduction  of  the  terms  "processes"  and  "actions"  reflect  our  belief  that  the  taxonomy  needs 
to  specify  human  performance  at  a  level  of  granularity  finer  than  the  subtask.  Thus  each  subtask 
would  be  analyzed  in  terms  of  the  inherent  processes  and  actions  required  of  the  hiunan 
participant(s)  in  order  for  the  subtask  to  be  performed  to  standard. 

The  lowest  levels  of  the  taxonomy  are 
measure  categories  and  user-defined  measures 
(indicators).  The  advocates  described  above 
win  play  an  important  role  at  these  levels 
and  at  operationally  defining  variables  for 
each  performance  dimension.  For  example, 
one  measure  category  of  the  performance 
dimension,  situational  awareness,  is  accuracy. 

A  potential  indicator  for  this  measure 
category  is  the  accuracy  of  SPOT  report 
locations.  Advocates  wiU  provide  input  as  to 
other  potential  indicators  and  indicator 
definitions.  It  is  our  view  that  the  measures 
are  a  function  of  purpose  and  hence  need  to 
be  defined  by  each  DIS  user  for  each  specific 
purpose.  The  advocates  will  establish  the 
parameters  for  their  community  and  a 
methodology  for  elaboration  of  measures 
appropriate  for  their  community. 

The  taxonomy  is  based  on  the  relationship  between  perceptual,  cognitive,  and  motor 
processes  which  defines  the  constructs  that  underlie  activities.  The  taxonomy  assumes  that 
processes  relate  to  observable  activities  and  behaviors  and  that  these  behaviors  may  be  viewed 
as  relating  to  performance  errors  which  target  the  sources  of  nonoptimal  performance.  In  turn, 
performance  dimensions  are  related  to  categories  of  measures  which  classify  operational  measures 
of  the  performance  dimensions.  Thus,  "time  to  detect  BLUFOR"  is  an  indicator  which  reflects 
the  measure  category  timeliness  and  the  performance  dimension  detection.  Finally,  indicators 
represent  a  specific  measure  of  performance.  Each  dimension  of  the  taxonomy  is  further 
described  below  and  a  strawman  of  the  DIS  TOOL  taxonomy  may  be  found  in  the  appendix. 


Figure  3.  DIS  TOOL  Human  Performance 
Taxonomy  Framework. 
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Processes.  Processes  give  a  broad  overview  of  the  underlying  variables  that  are  relevant 
to  unit  performance.  In  turn,  these  processes  relate  to  observable  behaviors  such  as  the 
^nchronization  of  tactical  movement  and  the  employment  of  mortars.  The  processes  providing 
the  framework  for  this  taxonomy  are  perceptual,  cognitive,  and  motor  in  nature.  In  turn,  these 
processes  may  be  broken  down  into  q)ecific  elements.  For  example,  an  important  aspect  of 
perceptual  processes  is  "collect  battlefield  information"  which  includes  processes  relevant  to 
monitoring  and  recognizing  important  battlefield  stimuli.  Cognitive  processes  include  those  steps 
involved  in  conducting  analyses  and  planning,  as  well  as  battlefield  management  and 
communication.  Finally,  motor  processes  involve  maneuver,  engagement  of  enemy,  and  the 
control  of  terrain.  These  processes  identify  the  aggregate  dynamics  at  work  in  the  combat 
environment  and  can  be  conceptualized  at  the  individual  or  unit  level. 

Activities.  From  the  standpoint  of  the  user,  processes  may  be  of  general  interest  but  they 
may  not  address  the  questions  being  asked.  Rather,  it  may  be  more  useful  to  examine  activities 
and  behaviors,  the  overt  manifestations  that  are  recorded  since  behaviors  are  often  seen  as  linking 
directly  with  performance.  To  ensure  that  unit  level  performance  is  measured,  the  activities 
identified  for  the  taxonomy  are  mostly  collective  in  nature  and  are  linked  to  key  tasks  associated 
with  the  CCFs.  For  example,  conducting  an  air  watch  is  an  important  component  of  CCF  17 
(take  passive  air  defense  measures)  and  fits  under  the  global  characterization  of  collecting 
battlefield  information.  Table  4  shows  each  of  the  processes  linked  to  a  set  of  relevant  battlefield 
activities  that  support  the  measurement  of  unit-level  mission  performance. 


Table  4 

DIS  TOOL  Processes  Linked  to  Battlefield  Activities 


PERCEPTUAL  Collecting  Battlefield  Information: 

Conduct  air/ground  watch,  receive  early  warning  signals,  focus/visualize 
the  battlefield. 

COGNITIVE  Conduct  Analysis  and  Pre-Planning: 

Evaluate/integrate/develop/prepare  information,  plan  for  combat 
operations,  direct  and  lead  unit  during  preparation  for  the  battle. 

Conduct  Final  Planning  and  Battle  Management: 

Update  information,  rehearse,  air  watch,  deceive,  synchronize  tactical 
operations,  manage  fire  control  and  distribution,  integrate  direct  fire  with 
maneuver,  manage  air  space,  consolidate  and  reorganize. 

Send,  Receive,  Disseminate  Information: 

Send/receive/disseminate  intelligence  and  battlefield  reports,  manage 
information. 

MOTOR  Maneuver,  Engage  Enemy,  Control  Terrain: 

Control  system  input;  navigate;  employ  direct  fire,  mortars,  field  artillery 
and  close  air  support;  cover  and  conceal;  disperse. 
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Error  Sources.  Error  sources  are  intended  to  serve  as  a  guide  to  future  DIS  TOOL 
users  wanting  to  expand  the  taxonomy  within  a  given  process.  The  error  sources  identified  for 
perceptual  processes  are  sensitivity  and  focus.  These  are  the  domains  under  which  performance 
errors  are  anticipated  for  perceptual  processes  and  should  provide  developers  with  important  cues 
as  to  where  performance  decrements  occur  and  the  types  of  indicators  which  will  target  those 
decrements.  For  instance,  a  training  developer  may  have  additional  "collect  battlefield 
information"  activities  that  he/she  desires  to  train.  While  this  might  require  the  development  of 
new  performance  measures,  the  error  sources  teU  the  developer  the  most  likely  origin  of 
nonoptimal  behavior.  In  the  case  of  collecting  battlefield  information,  the  error  sources  indicate 
that  soldiers  are  most  likely  to  err  in  terms  of  failing  to  detect,  incorrectly  detecting,  and  failing 
to  share  attention  optimally  between  various  sources  of  input. 

Performance  Dimensions.  DIS  users  usually  have  a  general  idea  of  the  types  of 
performance  variables  that  he  or  she  may  want  to  examine  such  as  "sensitivity  to  detection",  etc. 
Performance  dimensions  provide  DIS  users  with  broad  categories  of  potential  performance 
assessment  dimensions  which  can  be  measured  in  a  variety  of  ways.  For  instance,  the  performance 
dimensions  for  collecting  battlefield  information  are  detection  and  situational  awareness.  These 
were  identified  as  common  issues  for  many  of  the  DIS  users  at  the  Fort  Knox  Mounted  Warfare 
Test  Bed  (MWTB)  facility.  Both  detection  and  situational  awareness  are  vulnerable  to  sensitivity 
and  focus  errors  as  a  result  of  collecting  battlefield  information.  As  the  taxonomy  is  expanded, 
it  is  likely  that  additional  performance  dimensions  will  be  identified.  Community  advocates  and 
developers  should  ensure  that  new  performance  dimensions  link  to  key  components  of  the 
taxonomy  such  as  error  sources  and  activities  for  a  particular  process. 

Measure  Categories.  Measure  categories  classify  outputs  related  to  measures  of 
performance  including  behaviors  associated  with  timeliness,  completeness,  accuracy,  and 
operational  perspective.  Table  5  shows  each  category  and  its  definition.  These  categories  provide 
a  framework  for  categorizing  performance  in  terms  of  specific  criteria  such  as  accuracy  of 
performance.  An  important  aspect  of  these  measure  categories  is  that  they  are  constant  across 
performance  dimensions.  This  consistency  provides  DIS  users  with  a  "common  denominator"  with 
which  to  fi'ame  research  findings  for  test  issues  that  span  across  different  processes. 

Table  5 

DIS  TOOL  Measure  Categories  Defined 


ACCURACY  The  extent  to  which  misperception  of  a  stimulus  event  or  information  content 
result  in  an  incorrect  response. 

TIMELINESS  The  extent  to  which  reaction  time  to  a  stimulus  is  optimal  with  regard  to  METT- 
T,  SALUTE  (size,  activity,  location,  unit,  time,  equipment  and  5  Ws+H  (who, 
what,  when,  where,  why  and  how)). 

FREQUENCY  The  number  of  times  a  particular  behavior  occurs. 


OPERATIONAL  The  extent  to  which  information  is  accurately  placed  within  the  context  of  friendly/ 
PERSPECTIVE  enemy  force  operations,  areas  of  interest;  time/phasing  duration  of  the 
operation;  missions  of  higher,  lower  or  adjacent  units;  unit  contingency  plans  in 
place  or  in  development;  friendly  capabilities  related  to  operations;  enemy 
capabilities  related  to  operations. 
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Indicators.  Indicators,  such  as  "time  to  detect  OPFOR",  refer  to  specific  measures  of 
performance  which  are  grouped  by  the  broader  categories  described  above.  The  majority  of  the 
performance  measures  identified  in  the  taxonomy  have  been  successfully  operationalized  within 
a  DIS  environment.  The  current  framework  provides  a  fairly  comprehensive  mechanism  for 
categorizing  measures  of  performance  and  is  flexible  enough  to  allow  for  the  inclusion  of  new 
performance  measures  as  requirements  are  identified.  Developers  and  community  advocates  will 
play  a  heavy  role  in  the  identification  of  additional  performance  measures  for  each  measure 
category. 

Value/Utilitv.  Measures  must  demonstrate  utility,  that  is  they  must  show  their  value  to 
training,  etc.  by  their  ability  to  predict  success.  Thus,  while  not  specified  on  the  initial 
taxonomy,  the  final  taxonomy  will  evolve  to  eventually  identify  measmres  that  have  demonstrated 
predictive  validity  in  the  past. 

Utility  of  Taxonomy 

The  DIS  TOOL  taxonomy  will  relate  CCFs  to  unit  level  processes,  etc.  which  can  then 
be  translated  into  measures  of  performance.  For  instance,  a  trainer  could  relate  CCFs  to  a  DIS 
effort  by  specifying  exercises  that  incorporate  or  require  the  incorporation  of  several  CCFs 
simultaneously.  Thus,  the  DIS  exercise  could  examine  the  joint  influence  of  several  CCFs  and 
suggest  ways  to  measure  their  influence. 

The  DIS  TOOL  taxonomy  will  show  utility  as  it  relates  to  the  needs  of  the  Army  and 
provide  information  regarding  the  perceptual  and  cognitive  variables  deemed  important  for 
performance  measurement.  In  a  general  sense,  processes  are  related  to  activities  and  behaviors, 
which  in  turn  are  related  to  sources  of  error,  performance  dimensions,  categories  of  measures, 
indicators,  and  utility/value.  Thus,  these  different  structures  are  linked  to  each  other  in  a  way 
which  provides  the  user  a  mechanism  for  "tracing"  through  the  taxonomy  (from  processes  to 
indicators,  for  example).  The  importance  of  this  taxonomy  is  that  these  strucmres  link  to  each 
other  and  relate  general  performance  dimensions  and  processes  to  indicators.  Further,  the  DIS 
TOOL  taxonomy  specifies  important  links  between  various  aspects  of  human  performance  in  a 
way  not  yet  provided  to  DIS  users. 
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IV.  Future  Development  of  PIS  TOOL 


As  noted  earlier,  the  technical  approach  for  DIS  TOOL  has  been  established  but  more 
work  is  needed  to  develop  a  functioning  and  tested  tool.  The  completion  of  several  tasks 
characterize  the  progress  to  date.  First,  a  performance  plan  was  written  specifying  a  complete 
methodological  approach  for  the  initial  effort  (see  BDM  Federal,  1993).  Second,  community 
advocates  for  combat  development,  CGF  development,  training  development,  materiel 
development  and  test  and  evaluation  development  were  appointed  to  represent  their  respective 
communities.  Third,  the  development  of  a  DIS  TOOL  taxonomy  designed  to  reflect  processes 
and  behaviors  as  weU  as  related  performance  dimensions,  measures,  and  indicators  was  begun 
and  is  described  in  this  report.  Finally,  a  companion  report  describes  the  initial  development  of 
a  decision  taxonomy  based  upon  simulation  modeling  techniques  that  emulate  human  reasoning 
and  deciaon  making  (Weaver  &  Mullen,  in  preparation).  These  two  reports,  reflect  the  initial 
separate  approaches  to  achieving  the  technical  objectives  of  the  DIS  TOOL  effort.  The  largest 
task  remaining  is  the  marrying  of  the  two  approaches,  resulting  in  a  software  implementation 
which  will  demonstrate  the  utility  of  DIS  TOOL  in  satisfying  both  technical  objectives:  (a)  To 
design  and  execute  procedures  for  developing  a  taxonomy  of  human  performance  variables, 
including  perceptual  and  cognitive,  that  may  be  used  to  identify  significant  dimensions  that  need 
to  be  measured  to  enable  accurate  unit  performance  assessment  in  the  DIS  environment;  and  (b) 
to  demonstrate  the  utility  of  the  product  in  controlling  CGF  behavior  and  analyzing  unit 
performance.  The  union  of  these  two  objectives  for  the  final  product  is  contingent  upon  the 
completion  of  several  tasks  for  the  initial  effort.  Those  tasks  related  to  the  DIS  TOOL  taxonomy 
of  human  performance  are  discussed  below;  while,  tasks  related  to  the  CGF  decision  taxonomy 
are  discussed  in  Weaver  and  Mullen  (in  preparation). 

A.  Immediate  Tasks  Requiring  Further  Development 

Complete  the  link  of  missions  and  CCFs  to  taxonomy.  The  development  of  any 
taxonomy  requires  the  specification  of  the  boundary  conditions  for  the  problem  space  in  which 
the  taxonomy  will  be  developed.  The  boundary  conditions  for  DIS  TOOL  are  mission,  CCFs, 
and  unit  organization.  For  the  initial  effort,  two  missions.  Deliberate  Attack  and  Hasty  Defense, 
were  selected.  These  mission  encompass  a  broad  range  of  CCFs  as  well  as  represent  typical 
tactical  situations  that  can  be  supported  in  DIS. 

To  further  constrain  the  problem  space,  three  CCFS  were  selected  for  the  initial  analysis, 
CCF  6  (Engage  Enemy  with  Direct  Fire  and  Maneuver),  CCF  25  (Provide  Operations  Security), 
and  CCFs  16/17  (Take  Active/Passive  Air  Defense  Measures).  Through  these  three  CCFs,  other 
CCFs  such  as  CCF  23  (Provide  Countermobility)  and  CCF  5  (Conduct  Tactical  Movement)  were 
also  to  be  explored.  For  analysis  purposes,  the  focus  was  to  be  on  battlefield  activities 
surrounding  an  engagement  area  for  the  defensive  mission.  This  focus  was  believed  to  allow  for 
specification  of  tasks  from  battalion  through  individual  soldier.  As  part  of  the  current 
development  effort,  CCF  activities  were  linked  to  perceptual,  cognitive,  and  motor  processes  and 
drove  the  specification  of  the  remaining  taxonomy  components,  but  additional  development  and 
refinement  is  required. 

Within  this  problem  space,  a  list  of  doctrinal  references  and  source  materials  should  be 
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used  for  analysis.  The  CATT  database  should  provide  the  necessary  references  for  these  sources. 
These  sources  will  be  organized  into  a  matrix  linking  echelon  to  doctrinal  sources.  The  purpose 
of  this  matrix  is  to  gxiide  analysis  and  specification  of  subtasks,  decisions,  actions,  and  variables. 
This  matrix  will  be  particularly  important  in  yielding  sources  for  identifying  tasks  at  echelons  not 
currently  specified  in  the  CCFs  (i.e.,  below  platoon),  and  for  informing  the  process  of  variable 
identification  for  tasks  contained  in  the  CCFs. 

Further  involve  community  advocates.  The  initial  multiple  taxonomies  come  from  the 
need  of  the  DIS  TOOL  to  serve  a  variety  of  communities.  However,  it  is  recognized  that  the 
optimum  taxonomy  would  be  a  single  taxonomy  which  would  serve  each  of  the  communities 
equally  weU.  Since  it  is  simply  too  difficult  for  the  analyst  to  continually  keep  in  mind  the  needs 
of  potential  user  commimities  during  the  development  process,  advocates  from  five  primary 
communities;  combat  development,  training  development,  CGF  development,  materiel 
development,  and  test/evaluation  were  appointed.  Using  an  inductive  approach,  these  advocates 
need  to  be  closely  aligned  with  the  analysts  and  participate  substantially  in  brainstorming  sessions 
regarding  the  development  of  the  community-specific  aspects  of  the  taxonomy.  The  advocate  will 
serve  as  the  voice  and  eyes  of  their  respective  community  and  would  provide/collect  input 
focused  on  the  identification  of  issues  where  DIS  is  or  might  be  used,  the  type  of  analysis  issues 
that  normally  are  addressed,  the  variables  of  interest,  and  some  examples  of  measures  that  might 
or  would  be  used.  Finally,  the  community  advocates  would  review  the  draft  taxonomy,  make 
final  inputs,  and  be  responsible  for  determining  the  face  validity  of  the  taxonomy  from  the 
perspective  of  their  user  community. 

Complete  specification  of  performance  dimensions.  Based  on  the  linkage  of  mission 
and  CCFs  to  the  taxonomy  and  guided  by  community  advocate  input,  the  remaining  performance 
dimensions  and  associated  indicators  of  performance  must  be  specified.  Variable  definitions 
should  be  derived  using  both  an  empirical  and  community  approach.  The  empirical  approach  wUl 
use  selected  data  (Take  Home  Packages,  Replays,  Videos  of  the  After  Action  Reviews)  from  the 
NTC  Archive  and  wiU  center  on  analyzing  different  points  in  the  battle  to  determine  what 
performance  dimensions  or  variables  are  important.  Specifically,  the  advocates  wUl  look  at 
different  players  at  aU  echelon  levels  to  determine  what  is  important  to  their  performance  both 
in  terms  of  queuing  as  well  as  assessing.  In  addition,  this  process  wUl  allow  for  "validation"  of 
the  analysis  performed  through  the  brainstorming  sessions.  That  is,  if  the  initial  taxonomy  does 
not  include  performance  dimensions  necessary  to  imderstand  the  NTC  results  then  it  needs  to  be 
modified  or  expanded.  In  addition  to  NTC  data  sources,  test  plans  from  the  Director  of  Combat 
Development  (DCD)  at  Fort  Knox,  and  Lessons  Learned  information  from  Fort  Leavenworth  may 
also  be  accessed  and  reviewed.  The  purpose  in  each  case  is  to  help  specify  variables  of  interest 
and  concern,  and  to  help  identify  modifications  or  expansions  to  the  working  model  of  the 
taxonomy. 

The  results  of  the  empirical  analysis  will  provide  input  to  the  community  analysis.  In 
addition,  paper-based  demonstrations  should  be  conducted  to  elicit  SME  input  and  review.  This 
process  will  allow  for  iterative  revisions  and  improvements  and  should  result  in  a  taxonomy  with 
variable  specification  for  the  CCFs  under  analysis  and  a  methodology  for  defining  variables  for 
the  remaining  CCFs. 


18 


B.  Final  Taxonomy  Tasks 


Document  taxonomy  methodologies.  A  report  which  will  serve  as  the  blueprint  for 
the  completion  of  the  taxonomy  must  be  written.  The  report  must  lay  out  the  methodologies 
necessary  to  expand  the  analysis  to  other  missions,  CCPs,  and  echelons;  and  it  must  describe  the 
methodologies  for  combining  the  separate  taxonomic  approaches.  It  must  also  address  the 
process  of  identifying  performance  dimensions  and  measure  specifications.  In  this  regard,  it  will 
provide  procedures  for  identifying  clusters  of  tasks  that  share  measurement  commonalities  such 
that  variables  or  measures  that  apply  to  one  task  might  be  assumed  to  apply  to  all  tasks.  Thus, 
tasks  that  involve  a  certain  type  of  decision  or  that  employ  a  specific  behavior  such  as  firing  of 
a  weapon  would  be  identified  and  clustered  for  measurement  purposes.  Finally,  the  report  must 
describe  how  the  taxonomy  can  be  continuously  expanded  through  its  use.  That  is,  DIS  TOOL 
users  should  have  the  capability  to  add  performance  dimensions,  measure  categories,  and 
indicators  of  performance  to  the  taxonomy. 

Develop  DIS  TOOL  software  system.  Because  the  final  methodologies  adopted  for 
completing  the  DIS  TOOL  taxonomy  will  drive  the  software  system  to  a  certain  extent,  these  two 
tasks  should  be  completed  in  parallel.  The  product  of  the  DIS  TOOL  software  development 
effort  should  be  a  windows-based  automated  aid  that  wiU  be  useful  to  a  wide  variety  of  Army 
professionals  working  in  the  DIS  environment.  The  DIS  TOOL  wiU  provide  an  interactive  3D 
interface  that  aUows  structured  query  of  DIS  database  variables.  At  a  technical  level,  the  DIS 
TOOL  wiU  be  an  event-based,  object-oriented  program  that  is  unified  by  two  major  structures: 
a)  a  graphic  user's  interface;  and  2)  a  DIS  database  metastructure.  For  a  more  complete 
description  of  the  tasks  involved  in  developing  the  DIS  TOOL  software  system,  see  BDM  Federal 
(1993).  However,  the  foUowing  subtasks  organize  the  primary  activities  perceived  as  being 
involved  in  the  development  of  the  software  system: 

•  Specify  User  Requirements 

•  Formulate  System  Specifications 

•  Develop  Software  Preliminary  Design 

•  Develop  Detailed  Software  Design 

•  Develop  Software  Code 

•  Develop  System  Documentation 

Demonstrate  DIS  TOOL.  An  area  selected  for  major  importance  to  the  combat 
development  and  training  communities  should  be  selected  for  the  demonstration  effort.  Included 
in  the  demonstration  wUl  be  a  complete  taxonomy  down  to  the  measure  level  for  the  selected 
topic.  Supporting  the  demonstration  will  be  a  briefing  describing  use  of  the  tool  and  its 
advantages.  The  result  of  this  software  demonstration  wiU  be  proof  of  the  utility  of  the  taxonomy 
to  assess  unit  performance  and  control  CGFs. 
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Provide  final  documentation  of  effort.  A  final  report  should  be  prepared  describing  the 
taxonomy,  its  development,  results  of  the  demonstration,  and  examples  of  how  it  could  be  further 
applied  to  performance  measurement  and  CGF  development. 

Monitor  actual  use  of  PIS  TOOL  and  revise  accordingly.  The  use  of  DIS  TOOL  by 
a  variety  of  DIS  users  should  be  evaluated  for  a  specified  period  of  time  to  ensure  that  DIS 
TOOL  is  meeting  the  needs  of  its  users.  DIS  TOOL  users  should  be  interviewed  and 
recommendations  for  further  development/improvements  should  be  evaluated  by  the  respective 
community  advocates  and  analysts,  allowing  for  feedback  to  be  incorporated  into  the  ^stem  in 
a  timely  manner.  This  approach  is  a  key  aspect  of  the  iterative  development  process  and  is 
viewed  as  critical  to  the  successful  development  and  implementation  DIS  TOOL. 
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VI.  Appendix 


TOOL  Taxonomy 


DIS  TOOL  Taxonomy  Measure  Category  Definitions 


ACCURACY  The  extent  to  which  misperception  of  a  stimulus  event  or  information 
content  results  in  an  incorrect  response. 

TIMELINESS  The  extent  to  which  reaction  time  to  a  stimulus  is  optimal  with  regard  to 
METT-T,  SALUTE  (size,  activity,  location,  unit,  time,  equipment,  and  5  Ws 
+  H  (who,  what,  when,  where,  why,  &  how). 

FREQUENCY  The  number  of  times  a  particular  behavior  occurs. 

OPERATIONAL  The  extent  to  which  information  is  accurately  placed  within  the  context  of 

PERSPECTIVE  Mendly/enemy  force  operations.  Includes  information  concerning  areas  of 
operations;  areas  of  interest;  time/phasing/duration  of  the  operation;  missions 
of  higher,  lower,  or  adjacent  units;  unit  contingency  plans  in  place  or  in 
development;  friendly  capabilities  related  to  operations;  enemy  capabilities 
related  to  operations. 
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COLLECT  BATTLEFIELD  INFORMATION 


CANDIDATE  ACTIVITIES:  Conduct  air/ground  watch,  receive  early  warning 

signals,  focus/visuaHze  the  battlefield 

ERROR  SOURCES:  Sensitivity,  focus 

PERFORMANCE  DIMENSION:  Detection 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Percent  of  OPFOR  detected 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Time  to  detect  OPFOR 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Number  of  air-ground  attack  requests 


MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  Were  early  warning  signals  adequately  responded  to? 


PERFORMANCE  DIMENSION:  Situational  Awareness 


MEASURE  CATEGORY:  Accuracy 

POTENTIAL  INDICATOR:  Percent  of  Calls  For  Fire  with  correct  enemy  type 

MEASURE  CATEGORY:  Timeliness 

POTENTIAL  INDICATOR:  Mean  time  out  of  sector/misoriented 

MEASURE  CATEGORY:  Frequency 

POTENTIAL  INDICATOR:  Number  of  lases  to  friendly  vehicles 

MEASURE  CATEGORY:  Operational  Perspective 

POTENTIAL  INDICATOR:  Did  indirect  fire  planning  keep  pace  with  changes  in 

METT-T? 
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CONDUCT  ANALYSIS  AND  PRE-PLANNING 


CANDIDATE  ACTIVITIES:  Evaluate/integrate/develop/prepare  information,  plan  for 

combat  operations,  direct  and  lead  unit  during 
preparation  for  the  battle 

ERROR  SOURCES:  Interpretation,  integration,  sequence,  logic,  reconceptua¬ 
lization,  association,  conception,  prioritization,  focus, 
recognition,  reaction,  testing 

PERFORMANCE  DIMENSION:  Analysis 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Accuracy  of  IPB  Overlays/Templates 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Time  to  develop  IPB  Overlays/Templates 

MEASURE  CATEGORY:  Frequency 

POTENTIAL  INDICATOR:  Number  of  times  IPB  Overlays/Templates  updated 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  Was  translation  of  Brigade  OPORD  to  Commander's 

planning  guidance  accurate? 

PERFORMANCE  DIMENSION:  Pre-planning 


MEASURE  CATEGORY:  Accuracy 

POTENTIAL  INDICATOR:  Difference  between  planned  unit  dispersion  distance 

and  actual  dispersion  distance 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Time  to  develop  synchronized  OPORD 

MEASURE  CATEGORY:  Frequency 

POTENTIAL  INDICATOR:  Number  of  rehearsals  conducted  to  reach  desired 

outcome 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  Was  the  OPORD  understood  by  the  participants? 
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COGNITIVE  PROCESSES 


CONDUCT  FINAL  PLANNING  AND  BATTLE  MANAGEMENT 


CANDIDATE  ACTIVITIES:  Update  Information,  Rehearse,  Air  Watch,  Deceive, 

Synchronize  Tactical  Operations,  Manage  Fire  Control 
and  Distribution,  Integrate  Direct  Fire  with  Maneuver, 
Manage  Air  Space,  Consolidate  and  Reorganize 

ERROR  SOURCES:  Interpretation,  integration,  sequence,  logic,  reconceptua¬ 
lization,  association,  conception,  prioritization,  focus, 
recognition,  reaction,  testing 


PERFORMANCE  DIMENSION:  Final  Planning 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Accuracy  of  situational  template  updates 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Time  from  reception  of  OPORD  to  REDCON-1 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Nmnber  of  requests  for  FRAGO  clarification 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  Did  reconnaissance  effort  confirm  IPB? 


PERFORMANCE  DIMENSION:  Battle  Management 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Loss  exchange  ratio 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Time  to  reach  objectives 

MEASURE  CATEGORY:  Frequency 

POTENTIAL  INDICATOR:  Number  of  counterattacking  companies  engaging 

OPFOR 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  To  what  extent  did  the  Battalion  meet  the  Brigade 

Commander's  intent? 


NITIVE  PROCESSES 


CANDIDATE  ACTIVITIES:  SendSReceiveNDisseminate  intelligence  and  battlefield 

reports,  Manage  Information 

ERROR  SOURCES:  INTERPRETATION,  INTEGRATION,  ASSOOATION, 

CONCEPTION,  PRIORITIZATION,  FOCUS, 
RECOGNITION 


PERFORMANCE  DIMENSION:  Receive  and  Transmit  Information 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Consistency  of  relayed  INTEL  reports 

MEASURE  CATEGORY:  Timeliness 

POTENTIAL  INDICATOR:  Elapsed  time  from  artillery  impact  to  receipt  of  SHELL 

report  by  PLT  LDR/CO  CDR/BN  TOC. 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Number  of  reports  transmitted,  by  report  type 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  How  did  the  reception  and  transmission  of  enemy/ 

friendly  information  help/hinder  the  synchronization  of 
tactical  operations? 


PERFORMANCE  DIMENSION:  Workload 


MEASURE  CATEGORY:  Accuracy 

POTENTIAL  INDICATOR:  Number  of  reports  transmitted  with  omitted  information, 

by  report  type 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Mean  time  to  relay  reports  by  report  type 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Percent  duplicate  reports  received 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  How  did  the  management  of  information  affect 

individual  and  collective  battlefield  performance? 


26 


MOTOR  PROCESSES 

MANEUVER,  ENGAGE  ENEMY,  CONTROL  TERRAIN 


CANDIDATE  ACTIVITIES:  Control  system  input;  Navigate;  Employ  direct  fire, 

mortars,  field  artillery,  and  close  air  support;  cover  and 
conceal;  disperse 

ERROR  SOURCES:  Control,  Misinterpretation,  Reaction,  Location 


PERFORMANCE  DIMENSION:  Simple  Motor  Req)onses 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Distance  travelled 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Mean  velocity  while  moving 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Number  of  Rounds  Fired 

MEASURE  CATEGORY:  Operational  Perspective 

POTENTIAL  INDICATOR:  Were  troops  dispersed  in  an  optimal  manner? 


PERFORMANCE  DIMENSION:  Complex  Motor  Responses 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Average  target  counterfire  deployment  distance 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Time  between  threat  detection  to  counterfire 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Percent  of  firing  opportunities  taken 

MEASURE  CATEGORY:  Operational  Perfective 
POTENTIAL  INDICATOR:  Were  direct  fires  optimally  integrated  with  maneuvers? 
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SEND,  RECEIVE,  DISSEMINATE  INFORMATION 

CANDIDATE  ACTIVITIES:  Send\Receive\Disseminate  intelligence  and  battlefield 

reports,  Manage  Information 

ERROR  SOURCES:  INTERPRETATION,  INTEGRATION,  ASSOCIATION, 

CONCEPTION,  PRIORITIZATION,  FOCUS, 
RECOGNITION 

PERFORMANCE  DIMENSION:  Receive  and  Transmit  Information 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Consistency  of  relayed  INTEL  reports 

MEASURE  CATEGORY:  Timeliness 

POTENTIAL  INDICATOR:  Elapsed  time  from  artillery  impact  to  receipt  of  SHELL 

report  ,yby  PLT  LDR/CO  CDR/BN  TOC. 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Number  of  reports  transmitted,  by  report  type 


MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  How  did  the  reception  and  transmission  of  enemy/ 

friendly  information  help/hinder  the  synchronization  of 
tactical  operations? 


PERFORMANCE  DIMENSION:  Workload 


MEASURE  CATEGORY :  Accuracy 

POTENTIAL  INDICATOR:  Number  of  reports  transmitted  with  omitted  information, 

by  report  type 

MEASURE  CATEGORY :  Timeliness 
POTENTIAL  INDICATOR:  Mean  time  to  relay  reports  by  report  type 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Percent  duplicate  reports  received 


MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  How  did  the  management  of  information  affect 

individual  and  collective  battlefield  performance? 
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MANEUVER,  ENGAGE  ENEMY,  CONTROL  TERRAIN 


CANDIDATE  ACTIVITIES:  Control  system  input;  Navigate;  Employ  direct  fire, 

mortars,  field  artillery,  and  close  air  support;  cover  and 
conceal;  disperse 

ERROR  SOURCES:  Control,  Misinterpretation,  Reaction,  Location 


PERFORMANCE  DIMENSION:  Simple  Motor  Responses 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Distance  travelled 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Mean  velocity  while  moving 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Number  of  Rounds  Fired 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  Were  troops  dispersed  in  an  optimal  manner? 


PERFORMANCE  DIMENSION:  Complex  Motor  Responses 


MEASURE  CATEGORY:  Accuracy 
POTENTIAL  INDICATOR:  Average  target  coimterfire  deployment  distance 

MEASURE  CATEGORY:  Timeliness 
POTENTIAL  INDICATOR:  Time  between  threat  detection  to  counterfire 

MEASURE  CATEGORY:  Frequency 
POTENTIAL  INDICATOR:  Percent  of  firing  opportimities  taken 

MEASURE  CATEGORY:  Operational  Perspective 
POTENTIAL  INDICATOR:  Were  direct  fires  optimally  integrated  with  maneuvers? 
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